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DETAILED ACTION 
Response to Arguments 

1 . Applicant's arguments filed 6 January 201 2 have been fully considered but they 
are not persuasive. 

In response to applicant's arguments that the given references do not teach 
"wherein the movement... of the audience," Page 18, lines 10-15, the examiner 
respectfully disagrees. Kondo teaches utilizing various provided information, such as 
the time of day, the season, date, temperature, location of the event, i.e. the 
environment of the audience, in order to estimate an audience state, (Para. 244, 245). 
Furthermore, this information is combined with auxiliary information that indicates the 
state of the event being shown to the audience. Therefore, the aforementioned 
limitation is taught by the combination of the given references. 



Claim Rejections - 35 USC § 103 

2. The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set 
forth in section 1 02 of this title, if the differences between the subject matter sought to be patented and 
the prior art are such that the subject matter as a whole would have been obvious at the time the 
invention was made to a person having ordinary skill in the art to which said subject matter pertains. 
Patentability shall not be negatived by the manner in which the invention was made. 

3. Claims 1,2,4, 5, 29-34, 55-58 are rejected under 35 U.S.C. 1 03(a) as being 
unpatentable over Tow et al. (US 7,266,771) in view of Takata et al. (US 6,256,400) in 
view of Okada (US 5,907,361) and further in view of Kondo et al. (US 2002/0073417). 
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Regarding claim 1 , Tow teaches an audience state estimation system 
(Fig. 1A) comprising: 

imaging device for imaging an audience and generating a video signal 
relative to the audience thus imaged, i.e. a camera (Col. 1 , lines 20-42; Col. 1 0, 
lines 46-57); 

movement amount detection device for detecting a movement amount of 
said audience based on said video signal, i.e. determining motion information 
from video frames (Col. 3, lines 20-35; Col. 6, lines 47-59; Col. 8, lines 14-65); 

wherein the movement amount detection device divides an area into 
blocks (Fig. 2, el. 201, 203, 205, 207, 209; Col. 8, lines 14-65), and 

calculates a movement vector for each of the divided blocks (Fig. 2, el. 
201 , 203, 205, 207, 209; Col. 8, lines 14-65); and 

an estimation device for estimating an audience state based on a 
comparison result of said movement amount and a predetermined reference 
level, i.e. using a motion information template that corresponds to clapping (Col. 
10, line 46-Col. 11, line 3). 

Tow does not clearly teach the movement amount is selected to estimate 
an audience state based on a contents provision state which indicates an 
environment condition of the audience; the movement amount detection device 
discriminates and extracts a pixel range which is a flesh-color area identifying 
flesh color from said video signal, wherein the blocks include a face block 
representing a face unit of the audience and a hand block representing a hand 
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unit of the audience, and block matching of a current image and a next or 
previous frame image is performed for each of the blocks, wherein the movement 
vector is the movement direction and the movement amount when a result of the 
block matching indicating images of the blocks are matched, wherein each of the 
divided blocks includes a plurality of pixels and each of the plurality of pixels 
identifies flesh color. 

Takata teaches a movement amount detection device discriminates and 
extracts a flesh-color area identifying flesh color from a video signal, i.e. face and 
hand regions are extracted based on the identification of a flesh color (Col. 1 1 , 
lines 42-55; Col. 12, lines 16-43; Col. 18, lines 1-20; Col. 25, lines 55-65), 

wherein the video signal is divided into blocks including a face block 
representing a face unit of the audience and a hand block representing a hand 
unit of the audience, i.e. a face region and a hand region are identified (Fig. 7, el. 
701 , 703; Col. 1 2, lines 9-34; Col. 1 7, lines 1 9-49), and 

block matching of a current image and a next or previous frame image is 
performed for each of the blocks (Col. 1 2, lines 45-60; Col. 1 3, lines 1 -39; Col. 
18, lines 20-56; Col. 26, lines 1-7, 50-67; Col. 27, lines 43-67), 

wherein the movement vector is the movement direction and the 
movement amount when a result of the block matching indicating images of the 
blocks are matched, i.e. the currently extracted region is matched to a previously 
extracted region and movement information is determined (Col. 12, lines 45-60; 
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Col. 1 3, lines 1 -39; Col. 1 8, lines 20-56; Col. 26, lines 1 -7, 50-67; Col. 27, lines 
43-67). 

Therefore, it would have been obvious to one of ordinary skill in the art at 
the time the invention was made to modify Tow to include the movement amount 
detection device discriminates and extracts a flesh-color area identifying flesh 
color from said video signal, wherein the blocks include a face block representing 
a face unit of the audience and a hand block representing a hand unit of the 
audience, and block matching of a current image and a next or previous frame 
image is performed for each of the blocks, wherein the movement vector is the 
movement direction and the movement amount when a result of the block 
matching indicating images of the blocks are matched, using the known method 
of determining movement vectors for different regions of a human body, as 
taught by Takata, in combination with the motion vector determination system of 
Tow, for the purpose of enabling the system to better determine the attitude of 
the audience. 

Tow in view of Takata does not clearly teach the movement amount is 
selected to estimate an audience state based on a contents provision state which 
indicates an environment condition of the audience; the movement amount 
detection device discriminates and extracts a pixel range, wherein each of the 
divided blocks includes a plurality of pixels, and each of the plurality of pixels 
identifies flesh color. 
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Okada teaches a movement amount detection device discriminates and 
extracts a pixel range which is a flesh-color area identifying flesh color from said 
video signal, i.e. an area is extracted based on the color of a face (Col. 7, lines 
16-40; Col. 8, lines 15-27), 

divides the extracted flesh-color area into blocks identifying flesh color 
(Col. 7, lines 44-67; Col. 8, lines 33-55), and 

calculating a movement vector for each of the divided blocks (Col. 8, lines 
55-59; Col. 9, lines 9-14, 40-52), 

wherein each of the divided blocks includes a plurality of pixels and each 
of the plurality of pixels identifies flesh color (Col. 7, lines 27-41 ; Col. 8, lines 15- 
55). 

Therefore, it would have been obvious to one of ordinary skill in the art at 
the time the invention was made to modify Tow in view of Takata's movement 
amount detection device to include extracting a pixel range which is a flesh-color 
area identifying flesh color from said video signal, dividing the flesh-color area 
into blocks, and wherein each of the divided blocks includes a plurality of pixels 
and each of the plurality of pixels identifies flesh color, using the known method 
of extracting an area of a frame based on the color of a face, as taught by 
Okada, in combination with the audience estimation system of Tow in view of 
Takata for the purpose of providing improved correlation between an extracted 
area of an image and a preceding image (Okada-Col. 3, lines 23-30). 
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Tow in view of Takata in view of Okada does not clearly teach the 
movement amount is selected to estimate an audience state based on a contents 
provision state which indicates an environment condition of the audience. 

Kondo teaches a movement amount is selected to estimate an audience 
state based on a contents provision state which indicates an environment 
condition of the audience, i.e. using provided information to determine an 
audience response (Para. 69, 70, 153, 165, 166, 175, 190, 191, 240, 244, 245). 

Therefore, it would have been obvious to one of ordinary skill in the art at 
the time the invention was made to modify Tow in view of Takata in view of 
Okada to include the movement amount is selected to estimate an audience 
state based on a contents provision state which indicates an environment 
condition of the audience, using the known method of utilizing auxiliary 
information provided with the video data to determine an audience response, as 
taught by Kondo, in combination with the audience estimation system of Tow in 
view of Takata in view of Okada, for the purpose of providing a better estimation 
of audience response. 

Regarding claim 2, Tow in view of Takata in view of Okada in view of 
Kondo teaches the movement amount detection device determines movement 
information of the imaged audience based on said video signal (Takata-Col. 12, 
lines 45-60; Col. 1 3, lines 1 -39; Col. 1 8, lines 20-56; Col. 26, lines 1 -7, 50-67; 
Col. 27, lines 43-67), and 
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using MPEG differential frames that store motion information in the form of 
motion vectors obtained by determining the difference between adjacent frames 
(Tow-Col. 6, lines 47-60; Col. 8, lines 14-65); and 

wherein an average movement amount showing an average of 
magnitudes of the movement vectors is set as the movement amount (Tow-Col. 
9, line55-Col. 10, line 33). 

Regarding claim 4, Tow in view of Takata in view of Okada in view of 
Kondo wherein said movement amount detection device determines movement 
vectors of the imaged audience based on said video signal (Takata-Col. 12, lines 
45-60; Col. 13, lines 1-39; Col. 18, lines 20-56; Col. 26, lines 1-7, 50-67; Col. 27, 
lines 43-67; Tow-Col. 6, lines 47-60; Col. 8, lines 14-65); and 

calculating an average movement amount showing an average of 
magnitudes of the movement vectors (Tow-Col. 9, line 55-Col. 10, line 33), and 

wherein a time macro movement amount is set as the movement amount 
of said audience, said time macro movement amount being an average of the 
average movement amounts in a time direction thereof, i.e. the motion vectors 
have a magnitude and direction over the time period of a frame or several frames 
(Tow-Col. 9, line 55-Col. 10, line 33). 

Regarding claim 5, Tow in view of Takata in view of Okada in view of 
Kondo teaches when said movement amount is larger than a predetermined 



Application/Control Number: 10/602,779 Page 9 

Art Unit: 2427 

level, said estimation device estimates said audience state to be in any one of 
states of beating time with the hands and of clapping, i.e. using a motion 
information template that corresponds to clapping (Tow-Col. 10, line 46-Col. 1 1 , 
line 3). 

Regarding claims 29, 30, 55-58, claims are analyzed with respect to claim 
1 . These are Markush claims that include estimating the audience state based 
on audio taken from the audience, data of which was assigned to the non-elected 
Group II of the Restriction/Election Requirement, and likewise all limitations 
dealing with audio will not be examined. 

Regarding claim 31 , claim is analyzed with respect to claim 1 . 

Regarding claim 32, claim is analyzed with respect to claim 2. 

Regarding claim 33, claim is analyzed with respect to claim 4. 

Regarding claim 34, claim is analyzed with respect to claim 5. 



4. Claims 6-8 and 35-36 are rejected under 35 U.S.C. 1 03(a) as being unpatentable 
over Tow in view of Takata in view of Okada in view of Tagawa et al. (US 7,373,209) 
and further in view of Kondo. 
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Regarding claim 6, Tow teaches an audience state estimation system 
(Fig. 1A) comprising: 

imaging device for imaging an audience and generating a video signal 
relative to the audience thus imaged, i.e. a camera (Col. 1 , lines 20-42; Col. 1 0, 
lines 46-57); 

movement amount detection device for detecting a movement amount of 
said audience based on said video signal, i.e. determining motion information 
from video frames (Col. 3, lines 20-35; Col. 6, lines 47-59; Col. 8, lines 14-65); 

wherein the movement amount detection device divides an area into 
blocks (Fig. 2, el. 201, 203, 205, 207, 209; Col. 8, lines 14-65), and 

calculates a movement vector for each of the divided blocks (Fig. 2, el. 
201 , 203, 205, 207, 209; Col. 8, lines 14-65); and 

an estimation device for estimating an audience state based on a 
comparison result of the movement amount and a predetermined reference level, 
i.e. using a motion information template that corresponds to clapping (Col. 10, 
line46-Col. 11, line 3). 

Tow does not clearly teach the movement periodicity is selected to 
estimate an audience state based on a contents provision state which indicates 
an environment condition of the audience; a movement periodicity detection 
device discriminates and extracts a pixel range which is a flesh-color area 
identifying flesh color from said video signal, wherein the blocks include a face 
block representing a face unit of the audience and a hand block representing a 
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hand unit of the audience, and block matching of a current image and a next or 
previous frame image is performed for each of the blocks, wherein the movement 
vector is the movement direction and the movement amount when a result of the 
block matching indicating images of the blocks are matched, wherein each of the 
divided blocks includes a plurality of pixels and each of the plurality of pixels 
identifies flesh color. 

Takata teaches a movement amount detection device discriminates and 
extracts a flesh-color area identifying flesh color from a video signal, i.e. face and 
hand regions are extracted based on the identification of a flesh color (Col. 1 1 , 
lines 42-55; Col. 12, lines 16-43; Col. 18, lines 1-20; Col. 25, lines 55-65), 

wherein the video signal is divided into blocks including a face block 
representing a face unit of the audience and a hand block representing a hand 
unit of the audience, i.e. a face region and a hand region are identified (Fig. 7, el. 
701, 703; Col. 12, lines 9-34; Col. 17, lines 19-49), and 

block matching of a current image and a next or previous frame image is 
performed for each of the blocks (Col. 1 2, lines 45-60; Col. 1 3, lines 1 -39; Col. 
18, lines 20-56; Col. 26, lines 1-7, 50-67; Col. 27, lines 43-67), 

wherein the movement vector is the movement direction and the 
movement amount when a result of the block matching indicating images of the 
blocks are matched, i.e. the currently extracted region is matched to a previously 
extracted region and movement information is determined (Col. 12, lines 45-60; 
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Col. 1 3, lines 1 -39; Col. 1 8, lines 20-56; Col. 26, lines 1 -7, 50-67; Col. 27, lines 
43-67). 

Therefore, it would have been obvious to one of ordinary skill in the art at 
the time the invention was made to modify Tow to include the movement amount 
detection device discriminates and extracts a flesh-color area identifying flesh 
color from said video signal, wherein the blocks include a face block representing 
a face unit of the audience and a hand block representing a hand unit of the 
audience, and block matching of a current image and a next or previous frame 
image is performed for each of the blocks, wherein the movement vector is the 
movement direction and the movement amount when a result of the block 
matching indicating images of the blocks are matched, using the known method 
of determining movement vectors for different regions of a human body, as 
taught by Takata, in combination with the motion vector determination system of 
Tow, for the purpose of enabling the system to better determine the attitude of 
the audience. 

Tow in view of Takata does not clearly teach the movement periodicity is 
selected to estimate an audience state based on a contents provision state which 
indicates an environment condition of the audience; a movement periodicity 
detection device discriminates and extracts a pixel range, wherein each of the 
divided blocks includes a plurality of pixels and each of the plurality of pixels 
identifies flesh color. 
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Okada teaches a movement amount detection device discriminates and 
extracts a pixel range which is a flesh-color area identifying flesh color from said 
video signal, i.e. an area is extracted based on the color of a face (Col. 7, lines 
16-40; Col. 8, lines 15-27), 

divides the extracted flesh-color area into blocks identifying flesh color 
(Col. 7, lines 44-67; Col. 8, lines 33-55), and 

calculating a movement vector for each of the divided blocks (Col. 8, lines 
55-59; Col. 9, lines 9-14, 40-52), 

wherein each of the divided blocks includes a plurality of pixels and each 
of the plurality of pixels identifies flesh color (Col. 7, lines 27-41 ; Col. 8, lines 15- 
55). 

Therefore, it would have been obvious to one of ordinary skill in the art at 
the time the invention was made to modify Tow in view of Takata's movement 
amount detection device to include extracting a pixel range which is a flesh-color 
area identifying flesh color from said video signal, dividing the flesh-color area 
into blocks, and wherein each of the divided blocks includes a plurality of pixels 
and each of the plurality of pixels identifies flesh color, using the known method 
of extracting an area of a frame based on the color of a face, as taught by 
Okada, in combination with the audience estimation system of Tow in view of 
Takata for the purpose of providing improved correlation between an extracted 
area of an image and a preceding image (Okada-Col. 3, lines 23-30). 
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Tow in view of Takata in view of Okada does not clearly teach the 
movement periodicity is selected to estimate an audience state based on a 
contents provision state which indicates an environment condition of the 
audience; and a movement periodicity detection device for detecting movement 
periodicity of said audience based on said video signal. 

Tagawa teaches detecting periodicity based on an audio signal, i.e. 
identifying a periodicity of a rhythm or beat in music based on the peaks of an 
auto-correlation function of the audio (Col. 13, lines 5-49). 

Therefore, it would have been obvious to one of ordinary skill in the art at 
the time the invention was made to modify Tow in view of Takata in view of 
Okada to include a movement periodicity detection device for detecting 
movement periodicity of said audience based on said video signal, using the 
technique taught by Tagawa in combination with the motion vector system taught 
by Tow in view of Takata in view of Okada, although in different fields of 
endeavor would provide a predictable variation to the motion vector system and 
for the purpose of specifically identifying a type of motion for use in a video 
retrieval system. 

Tow in view of Takata in view of Okada in view of Tagawa does not clearly 
teach the movement periodicity is selected to estimate an audience state based 
on a contents provision state which indicates an environment condition of the 
audience. 
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Kondo teaches a movement amount is selected to estimate an audience 
state based on a contents provision state which indicates an environment 
condition of the audience, i.e. using auxiliary information to determine an 
audience response (Para. 69, 70, 153, 165, 166, 175, 190, 191). 

Therefore, it would have been obvious to one of ordinary skill in the art at 
the time the invention was made to modify Tow in view of Takata in view of 
Okada in view of Tagawa to include the movement periodicity is selected to 
estimate an audience state based on a contents provision state which indicates 
an environment condition of the audience, using the known method of utilizing 
auxiliary information provided with the video data to determine an audience 
response, as taught by Kondo, in combination with the audience estimation 
system of Tow in view of Takata in view of Okada in view of Tagawa, for the 
purpose of providing a better estimation of audience response. 

Regarding claim 7, Tow in view of Takata in view of Okada in view of 
Tagawa in view of Kondo teaches the movement periodicity detection device 
determines movement vectors of the imaged audience based on said video 
signal (Tow-Col. 6, lines 47-60; Col. 8, lines 14-65; Tagawa-Col. 13, lines 5-49), 

calculates an average movement amount showing an average of 
magnitudes of the movement vectors (Tow-Col. 9, line 55-Col. 10, line 33), and 

detects an autocorrelation maximum position of the average movement 
amount (Tow-Col. 1 0, line 57-Col. 1 1 , line 3; Tagawa-Col. 1 3, lines 5-49), and 
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wherein variance of the autocorrelation maximum position is set as said 
movement periodicity (Tagawa-Col. 13, lines 5-49). 



Regarding claim 8, Tow in view of Takata in view of Okada in view of 
Tagawa in view of Kondo teaches the variance is calculated using a signal in a 
frame range, said frame range being decided on the basis of the periodicity of 
said audience state to be estimated (Tow-Col. 9, lines 37-55; Col. 10, lines 45- 
67; Tagawa-Col. 13, lines 5-49). 



Regarding claim 35, claim is analyzed with respect to claim 6. 



Regarding claim 36, claim is analyzed with respect to claim 7. 



5. Claims 9, 1 0, and 37 are rejected under 35 U.S.C. 1 03(a) as being unpatentable 
over Tow in view of Takata in view of Okada in view of Tagawa in view of Kondo and 
further in view of Lu (US 5,550,928). 

Regarding claim 9, Tow in view of Takata in view of Okada in view of 
Tagawa in view of Kondo teaches all elements of claim 6. 

Tow in view of Takata in view of Okada in view of Tagawa in view of 
Kondo further teaches a movement periodicity (Tow-Col. 9, line 55-Col. 10, line 
33; Tagawa-Col. 13, lines 5-49). 
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Tow in view of Takata in view of Okada in view of Tagawa in view of 
Kondo does not clearly teach a ratio of low-frequency component in the average 
movement amount is set as said movement periodicity. 

Lu teaches subjecting an image to low-pass filtering (Col. 1 1 , line 48-Col. 
12, line 13). 

Therefore, it would have been obvious to one of ordinary skill in the art at 
the time the invention was made to modify Tow in view of Takata in view of 
Okada in view of Tagawa in view of Kondo to include a ratio of low-frequency 
component in the average movement amount is set as said movement 
periodicity, using the low-pass filtering technique of Lu in combination with the 
movement periodicity determining technique of Tow in view of Takata in view of 
Okada in view of Tagawa in view of Kondo for the purpose of removing 
extraneous image noise thereby providing a more accurate movement periodicity 
(Lu-Col. 11, lines 63-67). 

Regarding claim 1 0, Tow in view of Takata in view of Okada in view of 
Tagawa in view of Kondo in view of Lu teaches a frequency range of the low- 
frequency component is decided according to the periodicity of the said average 
movement amount transformed to a frequency region to be detected, i.e. 
identifying the rate and rhythm of clapping (Tow-Col. 9, line 55-Col. 10, line 33; 
Tagawa-Col. 1 3, lines 5-49; Lu-Col. 1 1 , line 48-Col. 1 2, line 1 3). 
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Regarding claim 37, claim is analyzed with respect to claim 9. 

6. Claims 1 1 and 38 are rejected under 35 U.S.C. 103(a) as being unpatentable 
over Tow in view of Takata in view of Okada in view of Tagawa in view of Kondo and 
further in view of Stevens (WO 91/03912). 

Regarding claim 1 1 , Tow in view of Takata in view of Okada in view of 
Tagawa in view of Kondo teaches all elements of claim 6. 

Tow in view of Takata in view of Okada in view of Tagawa in view of 
Kondo teaches determining the periodicity, rate, and rhythm of a set of motion 
vectors (Tow-Col. 10, line46-Col. 11, line 3; Tagawa-Col. 13, lines 5-49). 

Tow in view of Takata in view of Okada in view of Tagawa in view of 
Kondo does not clearly teach the estimation device estimates said audience 
state to be in a state of beating time with the hands when said movement 
periodicity is larger than a predetermined level, and estimates said audience 
state to be in a state of clapping when said movement periodicity is not larger 
than said predetermined level. 

Stevens teaches estimating a person to be in a state of beating time with 
the hands when said movement periodicity is larger than a predetermined level, 
and estimates a person to be in a state of clapping when said movement 
periodicity is not larger than said predetermined level, i.e. when a first loud or 
sharp sound is received a counter is started; when the counter reaches a 
predetermined level without the system receiving another loud or sharp sound, 
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the system determines that the periodicity of the first and a future second sound 
would be too large to be clapping; when a second loud or sharp sound is 
received before the counter reaches the predetermined level, the system 
determines that the periodicity of the two sounds is within a sufficient range to be 
clapping (Abstract). 

Therefore, it would have been obvious to one of ordinary skill in the art at 
the time the invention was made to modify Tow in view of Takata in view of 
Okada in view of Tagawa in view of Kondo to include the estimation device 
estimates said audience state to be in a state of beating time with the hands 
when said movement periodicity is larger than a predetermined level, and 
estimates said audience state to be in a state of clapping when said movement 
periodicity is not larger than said predetermined level, using the technique taught 
by Stevens in combination with the motion vector system taught by Tow in view 
of Takata in view of Okada in view of Tagawa in view of Kondo, although in 
different fields of endeavor would provide a predictable variation to the motion 
vector system and for the purpose of specifically identifying a type of motion for 
use in a video retrieval system. 

Regarding claim 38, claim is analyzed with respect to claim 1 1 . 
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Conclusion 

2. THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time 
policy as set forth in 37 CFR 1 . 1 36(a). 

A shortened statutory period for reply to this final action is set to expire THREE 
MONTHS from the mailing date of this action. In the event a first reply is filed within 
TWO MONTHS of the mailing date of this final action and the advisory action is not 
mailed until after the end of the THREE-MONTH shortened statutory period, then the 
shortened statutory period will expire on the date the advisory action is mailed, and any 
extension fee pursuant to 37 CFR 1 .1 36(a) will be calculated from the mailing date of 
the advisory action. In no event, however, will the statutory period for reply expire later 
than SIX MONTHS from the mailing date of this final action. 

Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to JEREMY DUFFIELD whose telephone number is 
(571 )270-1 643. The examiner can normally be reached on Mon.-Fri. 8:00 A.M. -5:30 
P.M. EST. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Scott Beliveau can be reached on (571) 272-7343. The fax phone number 
for the organization where this application or proceeding is assigned is 571-273-8300. 
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Information regarding the status of an application may be obtained from the 
Patent Application Information Retrieval (PAIR) system. Status information for 
published applications may be obtained from either Private PAIR or Public PAIR. 
Status information for unpublished applications is available through Private PAIR only. 
For more information about the PAIR system, see http://pair-direct.uspto.gov. Should 
you have questions on access to the Private PAIR system, contact the Electronic 
Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a 
USPTO Customer Service Representative or access to the automated information 
system, call 800-786-91 99 (IN USA OR CANADA) or 571 -272-1 000. 
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Supervisory Patent Examiner, Art Unit 2427 



